This application relates to digital image processing methods and apparatus, and more particularly to adaptive filtering methods and apparatus.
Block-based image and video coding as in current coding standards, such as JPEG, H.261, H.263, MPEG4 and H.264/AVC, divide an image to be encoded into horizontal rows and vertical columns of non-overlapping blocks of pixels, as illustrated by FIG. 1. The blocks are typically square, having dimensions of 8 pixels×8 pixels. A color image is often separated into a luminance (Y) channel that represents brightness information in the image and two chrominance (U, V) channels that represent color information in the image. The image is then represented by luminance blocks and, depending on the image's chrominance format, a number of chrominance blocks. Each block is encoded by a transform, e.g., the discrete cosine transform (DCT), and each transformed block is quantized according to a quantization parameter (QP).
Quantization can result in visually annoying artifacts, especially for video coding at low bit-rates relative to the video resolution, such as 32 kilobits per second (kbps) and 64 kbps for common intermediate format (CIF) and quarter CIF (QCIF) resolutions, which are used today for video clips, video conferences, etc. displayed on mobile communication devices and other devices having limited computational and display resources and/or communicating on channels having limited bandwidth. Two common artifacts produced by the quantization of the DCT are blocking and ringing artifacts. The blocking artifact is seen as an unnatural discontinuity between pixel values of neighboring blocks. The ringing artifact is seen as high-frequency irregularities around edges of objects in an image.
Many image processing methods to reduce blocking and ringing have been suggested. An older review is M.-Y. Shen and C.-C. J. Kuo, “Review of Postprocessing Techniques for Compression Artifact Removal”, J. Visual Communication and Image Representation, Vol. 9, No. 1, pp. 2–14 (March 1998).
To reduce blocking artifacts, two-dimensional (2D) low-pass filtering of pixels on block boundaries of the decoded image(s) was suggested in H. C. Reeve III and Jae S. Lim, “Reduction of Blocking Effect in Image Coding”, Proc. ICASSP, pp. 1212–1215, Boston, Mass. (1983). The 2D space-invariant static filtering described in that Reeve paper reduces blocking artifacts but can also introduce blurring artifacts when true edges in the image are filtered.
To avoid blurring of true edges in the image and also to be computationally efficient, the amount of digital low-pass filtering may be controlled by table-lookup as described in U.S. Pat. No. 5,488,420 to G. Bjontegaard for “Cosmetics Filter for Smoothing Regenerated Pictures, e.g. after Signal Compressing for Transmission in a Narrowband Network”. Large differences between initial pixel values and filtered pixel values are seen as natural image structure, and thus filtering is weak so that the image is not blurred. Small pixel differences are seen as coding artifacts, and thus stronger filtering is allowed to remove the artifacts. Based on data from other equipment, the amount of filtering can be controlled by using additional filter tables. This patent modifies the output of a low-pass-filtered signal with the output of a table-lookup using the difference between a delayed input signal and the filtered signal as an index into the table, and different degrees of filtering are achieved only by providing additional tables.
To adapt the amount of filtering according to the DCT quantization, the QP from an H.261 decoder has been used as described in U.S. Pat. No. 5,367,385 to X. Yuan for “Method and Apparatus for Processing Block Coded Image Data to Reduce Boundary Artifacts between Adjacent Image Blocks”. The QP scales the amount of low-pass versus all-pass filtering performed, thereby allowing stronger low-pass filtering for larger quantization. This patent selects low-pass and all-pass filter coefficients to reduce the differences in pixel values between boundary pixels of neighboring blocks, and the selection is determined by intra/inter-block encoding, the number of non-zero a.c. coefficients in the DCT, output pixel position in the block, and the direction of image gradient along the block boundary. The QP is used to weight the relative parts of the low-pass and all-pass filter responses in the filter output.
A combined de-blocking and de-ringing filter was proposed in ITU-T Recommendation H.263 Appendix III: Examples for H.263 Encoder/Decoder Implementations (June 2000). The proposed filter used filter strengths on block boundaries that were different from filter strengths inside blocks, allowing for stronger filtering at block boundaries than inside blocks. This was achieved by using a metric that used different constants when computing the output values of block boundary pixels versus the output values of pixels inside the block boundary. The metric also included the QP.
These and most other current algorithms handle de-blocking and de-ringing artifacts separately. This requires filtering in two steps to handle both artifacts, e.g., first process a decoded image with a de-blocking filter to remove artifacts on block boundaries, and then apply a de-ringing filter to remove ringing artifacts. Such double filtering can have a negative impact on computational complexity and memory consumption, which are parameters of particular importance in many devices, such as mobile communication devices.
Moreover, removal of blocking and ringing artifacts can add visually annoying blurring artifacts as described above. It is thus important to be careful with strong image features that likely are natural image features and not coding artifacts.
An adaptive non-linear filter can also be configured to increase sharpness of true image details, limit overshoots near sharp edges, and attenuate coding artifacts, as described in G. Scognamiglio et al., “Enhancement of Coded Video Sequences via an Adaptive Nonlinear Post-processing”, Signal Processing: Image Communication, vol. 18, no. 2, pp. 127–139, Elsevier B.V. (February 2003). A non-separable, two-dimensional, 5-tap filter is controlled by rational functions, one in the horizontal direction and one in the vertical direction. The rational functions are adapted based on measures of local variance and blockiness and ringing.
Most current algorithms consider sharpening separately from de-blocking/de-ringing. This can adversely impact computational complexity and memory consumption. In addition, increasing the sharpness of an image can add visually annoying noise and/or coding artifacts. Thus when sharpening, it is important to be careful with weak image features that can be coding artifacts or noise and not natural image features. In addition, a sharpening filter should have low complexity and low memory consumption, facilitating its use in resource-constrained environments, such as portable devices like mobile terminals.